A new family dependent interaction in Tevatron top dilepton candidate events ? 
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New family dependent fermionic interactions have been conjectured in several extensions of the 
Standard Model that range from Supersymmetry to composite theory up to flavor interactions. 
Strong constraints on these theoretical scenarios can be derived from light fermion phenomenology 
and from B-mesons studies. Corresponding constrains on the top quark sector are, on the other 
hand, rather week. Tevatron data, on top quark pair production and decay in dilepton channel, may 
suggest some deviation from the Standard Model expectations. Such a deviation can be successfully 
re- interpreted in terms of an exotic top decay that can arise in several theories beyond the Standard 
Model. Further investigations at present and future colliders will provide crucial tests on the models 
under discussion. 
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I. INTRODUCTION 

After the discovery of the top quark at Fermilab Teva- 
tron Collider P, experimental attention has been 
turned on the examination of its production mechanisms 
and decay properties. Within the Standard Model (SM), 
the top quark production cross section is evaluated with 
an uncertainty that is of the order of ~ 15% and top 
quarks are assumed to decay to a. W boson and a b 
quark almost 100% of the time. In the fermion sector 
of the SM, plenty of experimental data, have proved that 
interactions of quarks and leptons, with gauge bosons, 
are correctly described by the SU{?>)c x SU{2)l x U{1)y 
gauge theory At tree-level, SM neutral interactions 
are diagonal in flavor space, however, due to the genera- 
tion mixing, that appears in the charged currents, flavor 
changing neutral currents (FCNC) can arise at loop level. 
The fact that FCNC i?-meson decays have been already 
detected, at rates consistent with the SM Q, represents 
a great success for the model itself. Current bounds, on 
rare decays of light quarks and leptons, impose further 
constraints on new sources of FCNC transitions. How- 
ever, SM predictions for related processes, involving top 
quarks, are strongly suppressed, although corresponding 
experimental bounds are rather weak. 

Due to its exceedingly heavy mass, it is reasonable to 
expect the top to be more related to new physics than 
other fermions. Top quark physics may then serve as an 
important window for probing physics beyond the SM. 
A significant deviation from SM prediction would then 
indicate either a novel production mechanism or a novel 
decay channel that is the subject of the present paper. 
As matter of fact, it could be even possible that such 
effects may be observed at present Tevatron collider ex- 
periments (CDF-II, DO) or in the next generation collider 
experiments at LHC 's^ or later on at the next Linear Col- 



lider |g . FCNC top quark decays, such as t ^ 07, t — > eg, 
t cZ and t c4> have been studied, for some time, as 
possible signals of physics beyond the SM Jt'I. The ex- 
tended models, that we are interested to discuss, in the 
present article, include both top-charm (TC) transitions 
and /i — T violating (MTV) interactions, whose combined 
effect can induce top quark decays such as i — + cr/i or 
t uTii. Such processes may arise either through a 
scalar bosons, with family-dependent couplings, or by di- 
rect contact interactions. We shall refer to t c{u)t^ 
as double flavor- violating (DFV) top decays. 

II. THEORETICAL MOTIVATIONS 

Past and present Tevatron top analysis have been per- 
formed, always assuming, a dominant SM branching ratio 
{Br{t hW) ~ 1). The possibility that exotic top quark 
decay modes could affect the dilepton top quark events 
[pp ti W^bW^b b£^ vcM^ i7i) have not been con- 
sidered yet. In this paper, we are interested in studying 
how the dilepton signature, could be affected when one 
includes additional top decay channels such as: t crfx 
or t ^ UTji. We will argue later that such an effect could 
have already shown up and may be available in the top 
quark data published by the CDF collaboration 

Since in the SM, the branching ratio of top decaying 
into c{u)^T is extremely suppressed |0, ^|, then the 
study of this decay channels may be an excellent way for 
searching for physics beyond the SM. In the next sections 
we will discuss the expected branching ratios arising in 
several and well motivated extensions of the SM, which 
have been proposed to tackle some open issues. 

Among these theories, we shall discuss first Supersym- 
metry (SUSY), which is studied mainly in connection 
with a possible solution of the hierarchy problem [T^ . 
Its minimal incarnation, the Minimal Supersymmetric 
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extension of the SM (MSSM), has well know attractive 
features, such as allowing gauge coupling unification, and 
explaining electroweak symmetry breaking (EWSB) as a 
radiative effect, driven by the large top quark Yukawa 
coupling. In addition SUSY also predicts tau-bottom 
Yukawa unification and provides a dark matter candi- 
date. Within the context of SUSY models, it is possible 
to induce FCNC Higgs interactions, which could mediate 
the MTV top decay t c^t 

The resulting structure of the Higgs- Yukawa sector in 
MSSM, corrected by loop effects, actually shares similar- 
ities with the general version of the two-Higgs doublet 
model (THDM-III), whose predictions for DFV top de- 
cays will be discussed, for comparison, with other mod- 
els. The Yukawa structure of the THDM-III relies on 
the assumption of a specific texture, for fermion mass 
matrices, which makes possible to derive a pattern for 
TC and MTV interactions, which is possible to constrain 
with low-energy physics experime nts, making also avail- 
able predictions at higher energy A particular suc- 
cessful ansazt, for the THDM-III, consists in assuming a 
four-texture for the mass matrices, which is able to re- 
produce quark and charged lepton masses, as well as the 
CKM mixing matrix. This approach can be considered as 
an starting point to address one of the most challenging 
problems of the SM, the flavor problem, namely, how to 
explain the observed pattern of quark and lepton masses 
and their mixing angles. 

Along the lines of flavor physics, we are also going 
to discuss models that attempt to explain the struc- 
ture of the fermion mass matrices by an approximate 
flavor symmetry. Models, with either Abelian or non- 
Abelian flavor symmetry, have been proposed in litera- 
ture The scale, associated with these models, could 
range from TeV up to the Grand Unification scale (GUT 
scale). Flavor Models, with a low scale flavor symmetry 
breaking, offer a possibility to observe direct effects of fla- 
vor physics, for instance through the detection of scalar 
particle associated with the mechanism of flavor symme- 
try breaking, the so called Flavon Fields. On the other 
hand, when the flavor symmetry is a global-type symme- 
try, its breaking leaves a light pseudo-goldstone bosons, 
the familons that could provide detectable signatures 
too ■ The effects of the flavons or familons could even 
be transmitted to the EWK Higgs sector, through mix- 
ing 1^, and could induce exotic Higgs bosons and top 
quark decays. 

Finally we shall also consider strongly interacting sce- 
narios [l2|. These models are interesting as they offer 
some clue about the nature of EWSB. As matter of fact, 
such models often predict new physics to manifest it- 
self in the top quark sector mainly because of the large 
top-quark mass. They can also arise because of the pres- 
ence of additional substructure layers, which could mani- 
fest themselves first for the heaviest SM fermion. Strong 
interaction scenarios could then induce exotic DFV top 



quark decays too. 

III. TC AND MTV MODELS 

A. Overview 

In order to search for effects of a new family-dependent 
interaction (FDI), in top quark decays, we will start 
considering first the case when such interaction is me- 
diated by a new boson (S*"). The case, where such 
decays occurs through an unsuppressed contact inter- 
action, will be discussed separately. The interactions 
of S*" is here described by an effective lagrangian, that 
parametrizes family-dependent couplings for quarks and 
leptons, q,lL^ji, in terms of form factors: -f^''^. For the 
top-charm and tau-muon couplings the lagrangian takes 
the following form: 

£ = S''i[FlPL+F^PB]c+S''f[FiPL+F'j,Pj,]fi+h.c. (1) 

In the present case, the top quark decay t —^ cfir will 
proceed through the 5° intermediate state, namely t 
cS° and then S" ^ ^J,T. 

B. TC and MTV in Minimal SUSY Models 

Within the context of the MSSM, we shall consider 
that the double flavor-violating (DFV) top quark de- 
cays t cfiT are mediated through TC and MTV Higgs 
interactions, which arise through loops involving gaug- 
inos and sfermions. In particular, lepton flavor viola- 
tion (LFV), experimentally observed in the neutrino sec- 
tor [13, could be transmitted from the SUSY breaking 
sector to sfermions, and then from the sfermions to the 
Higgs sector. In this way the tree-level type-II Higgs 
sector of MSSM becomes a type-Ill THDM. Thus, LFV 
Higgs interactions in MSSM, which induce decays such 
as (j>i = h, H, A), are produced by loops involv- 

ing sleptons and charginos or neutralinos [l^ ■ Moreover, 
because SUSY GUT's relate quarks and leptons, it is pos- 
sible that the large mixing, observed in the neutrino sec- 
tor, implies a large mixing in the up- type squarks, which 
in turn, could also induce the FCNC top decay t chP 
[20|. By combining these effects it is possible to have 
exotic top quark decay t —> ch^ , followed by h^^ ^t. 

We shall consider a minimal SUSY-FCNC schemes, 
where a large mi xing appears in the slepton and squark 
trilinear A-terms |20|| . In particular we will consider that 
the elements of A-terms satisfy the conditions: A33 = 
Ao,A^3 = XuA'^, = xiA\^, where Xu I are unknown 
0(1) parameters, Aq denotes an universal SUSY breaking 
trilinear term and the remaining A-entries are assumed to 
vanish. After diagonalizing the resulting sfermion mass 
matrices, and writing the interactions in terms of mass 
eigenstates, one can evaluate the TC/MTV top and Higgs 
decays. The branching ratios will be sensitive to the 
values of Xu,i, since these parameters affects both the 
sfermion mass-eigenvalues (that enter the loop integrals) 
and the TC/MTV couplings in the i—c and fl — f sectors. 
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C. TC and MTV interactions in the THDM-III 

The THDM-III, based on the assumption of a specific 
four texture for the fermion rnass matrices, has been ex- 
tensively discussed in Ref. ^ISi]- In that paper we de- 
rived the resulting pattern for TC and MTV Higgs in- 
teractions, which generalizes the Cheng-Sher ansazt [2l|: 
namely the phases allowed by the hermitian four-texture 
ansazt were also included in the analysis. We derived also 
constraints on the model parameters, by comparing low- 
energy physics processes (mainly: /i ^ e -I- 7, r — > 3^, 
e — conversion and rare i?-decays). This allowed us 
to make predictions on the detectability of MTV Higgs 
decays at present and higher-energy colliders 22]. 

In this model, the couplings, of the lightest Higgs boson 
{h^) with fermions of different flavor, can be written as 
follows: 



Chuf, = ^^^-^x^,m.hfoh° (2) 

where m^j corresponds to the masses of the i and j 
fermions, v denotes the scale of EWSB and Xij are un- 
known 0(1) coefficients. The factor rjh depends on the 
fermion type. For u-type quarks its value is: rjh = 
— cos(/3 — a)/2-\/2sin/3, while for d-type quarks and lep- 
tons: ?7/i — cos(/3 — Q!)/2-\/2 cos/3, where a denotes the 
mixing angle in the CP-even Higgs sector. Therefore, 
the form factors for the couplings fifj are: Fl = Fj^ = 

D. Flavored bosons and TC/MTV top decays 

Models with a low flavor scale, which offer the possibil- 
ity to observe direct flavor physics effects, could also pre- 
dict large rates for the decay: t — > c/ir. One way to me- 
diate this decay can be provided by the scalar particle re- 
sponsible for the flavor symmetry breaking, the so called 
flavon fields In this case, we shall consider a model 

that contain a scalar particle that couples to fermions of 
different generations. A convenient parametrization, for 
its couplings, is provided by the Cheng-Sher ansazt [2l|. 
In order to maximize the effects, in the following analy- 
sis, we shall assume a scenario where the flavon field cou- 
ples predominantly to fermions of different generations. 
Such a patt ern can arise when fiavor symmetry is non- 
abelian |l5| . Then, we assume that the mass-eigenstate 
5° results form the mixing between a flavon fleld <I>^ and 
a Higgs boson The form factors for the coupling 

S°nfj are: 

Fl = Fr = ^rriim - (5^) sin^ -|- cos^%] (3) 

where ^ correspond to the masses of the fermion i and 
j, V denotes the v.e.v., that breaks the electroweak sym- 
metry, and ^ is the angle that parametrizes the mixing 
between the Higgs and the flavon flelds, in the scalar sec- 
tor. The factor Xij is again an 0(1) unknown coefficient. 



E. Strong Dynamics 

Furthermore, because the top quark is the heaviest 
SM fermion, it could offer some clue about the nature of 
EWSB, or could even be a composite state. In any case, 
it is likely that strong dynamics could induce rare top 
quark decays too. In this case, the strong dynamics could 
be associated with models of composite quarks and lep- 
tons and it is possible to have contact interactions, that 
can also mediate exotic transitions. The contact interac- 
tion for tcT^, including only a scalar or a pseudoscalar 
coupling, is given by: 

C = '^tVs cfT g ^ + h.c. (4) 

where C23 is an unknown coefficient that has been calcu- 
lated to be about mj/967r [i^ and F., = 1(75) for scalar 
(pseudoscalar) couplings. 

IV. RESULTS ON DFV TOP Br(t ^ cry) 

We are now interested in the evaluation of the double- 
flavor-violating top decay Branching Ratios, Br{t 
cfir), for all the models discussed above. When this de- 
cays are mediated by the scalar resonance S'^, we can 
factorize the branching ratio as follow: 

Br{t -> cr/i) = Br{t cS°) x Br{S° -> ^ir) (5) 

where Br{t cS°) = F(i ^ cS")/Tt. For the total 
top decay width, we will use the approximation: Fj = 
F(t -> bW+)sM and Br{S° fir) ^ F(S'0 ^ /iT)/Fs. 
At this point the total width of the scalar will depend 
only from the model. The decay widths for t — s- cS'° and 
— > /IT are then written in terms of form factors {Fl^h) 
as follows: 

F(t->c5°) = z^(i-!4)'^'[i^iP+i^An (6) 

r(5V) = + (7) 

an 

We find the following results for each of the model con- 
sidered. 

1. MSSM. In the case of the MSSM, the DFV decay 
t —^ cS'^ followed by S'^ —> is always kinematically 
allowed for 5° — . Then, as shown in table 1, the 
decay branching ratio Br{t —> chP) can be as large as 
lO"'^ — 10~^, over a large part of the SUSY parameter 
space, where the mass of the lightest Higgs boson is 
around 110 -f- IZOGeV .20). For a gluino mass: M~g = 
500 GeV, pseudo-scalar Higgs mass: — 300 GeV, 

SUSY parameters (m,At,^„) = (0.6,0.3,1.5) TeV, Xu = 
0.9, tan/3 = 5(10), one finds: Br{t ch^) = 4.0 x 
10^"* (lO^'^). Similarly, one can evaluate the decay h 
jiT, and finds a branching ratio of the order of 10^"' 
10~^ In fact, for a Bino-mass — 600 GcV, 

{m,n,Ai) = (0.6,0.3,0.3) TeV, xi = 0.9 and tan/3 = 
5(10) we get: Br(/i" -> ^lT) = 4.4 x 10-4(1.2 x 10"'*). 
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TABLE L Br{t ~* chP) for a sample set of Al-type inputs 
with (mO,^i,ylo) = (0.6, 0.3, 1.5) Tel/. The two numbers in 
each entry correspond to a; = (0.5, 0.9), respectively, tua and 
rrig are expressed in GeV . 



Therefore the branching ratio for the DFV top quark 
decay t — > c/ir reaches a value of the order 2 x 10"'' for 
tan /3 = 5(10), which is well bellow present top analysis 
sensitivity. 

2. The THDM-III. For the THDM-III we have that 
low energy physics implies that X23 is constrained to be of 
order 10~^, while X23 remains essentially unconstrained. 
Therefore, we can take it of order one. For rrih = 120 
GeV, and 5 < tan/3 < 50, a = /3 — 7r/3, it is possi- 
ble to obtain: X23 = O-l; ^-iid Br[hP /ir) only gets 
up to about 2 X 10^^, while for the TC top decay we 
find: Br{t ch°) ~ 0.1. Then, for DFV top decay, 
we calculate: Br{t cfJ-T) = 2 x 10^®. It should be 
pointed out that low values for Br{h^ fir) are ob- 
tained in this model, because its flavor structure forces 
the coefficients to satisfy the condition: X23 = X131 which 
then allows to use the rare decay /i — > 67 to constrain 
the parameter. When this relation no longer holds, it is 
possible to have: X23 — 1' ^^'^ then this makes possi- 
ble to obtain: Br{h^ /ir) ~ 4 x 10^^, and therefore: 
Br{t cjjT) = 4 X 10"^ 

3. Flavon-Higgs Mixing Model. The result for 
the branching ratio will depend on the mixing angle ^, 
introduced to parametrize the mixing between the Higgs 
and flavon fields. We shall assume that the lightest mass- 
eigenstate, which is denoted by S'^, has mass in the in- 
termediate range: mz < ms < nrit ~ rric. Then, in order 
to compute its branching ratio, we shall approximate it 
as: Br{S° ^t) = T{S° th)/T{S° bb). For a rea- 
sonable range of parameters, sin^ — 0.9, ms = 120 GeV, 
we find that Br(t cS°) -0.1, Br{S" fir) ~ 0.4, 
therefore Br{t cfir) ~ 0.04. As the parameters X23 
and ^ are essentially unconstrained, larger values for 
Br{t —^ c^t) around 0.1, are also achievable in this 
model. 

4. Strongly Interacting Model. In this scenario, 
there is a contact interaction that induces directly the 
decay t — > c/it, with a partial width given by: 

1 Tn ^ 



Since there are no constraints on such operators, one can 
even have A ~ mt, which then gives a branching fraction 
for DFV top decay of order of ~0. 1-^0. 2. A summary 
of all the results found is shown in Table II. 



IV. MTV INTERACTIONS AND tt hl+ ughr i7e 

A. Overview 

Within the SM, a top, with a mass above Wb thresh- 
old, is predicted to have a decay width dominated by 
the two-body process: t Wb {Br ~ 0.998) A 
tt pair will then decay via one of the following chan- 
nels, classified according to the final state: i) Dilepton, 
where both W decays are leptonic {£w), with 2 jets 
arising from the two 5-quark hadronization and miss- 
ing transverse energy (-^t) coming from the undetected 
neutrinos Hi]: Br(e, e) = Br(/Lt,/z) = Br(T, r) ~ 1/81, 
Br{e,iJ,) = Br{e,T) = Br{^,T) ~ 2/81; ii) Lepton+jets, 
where one W decays leptonically and the other one into 
quarks, with 4 jets and^T^ Br{e+jets) ~ Br{p+jets) — 
Br{T + jets) ~ 12/81; in) All-hadronic, where both the 
M^'s decay into quarks with 6 jets and no associated I^t: 
Br (jets) ~ 36/81. A r, coming from W, will then decay 
either in eVf,Vr or fiV^^v-r (ti) or in a narrow jet (t/j), 
with branching ratios that are: ifJ-~i^ fii^r) /^lotai ~ 
0.1736 ± 0.006, T^{e^yei^r)lTl^tai = 0.1784 ± 0.006 and 
r^(r jet)/V[^t^i ~ 0.64 [3|. Then r's coming from tt 
decay will give rise to the following final states: ^TThThbb, 
^TThTibb, ^T^TfiTibb, ^TiwThbb and ^T^wTibb. Further 
on, we will call dilepton only the following final states: 
ee + X, efi + X or /i/i + X. 

In both CDF Run 1 and Run 2 tt dilepton analysis, 
events were selected by requiring two opposite sign (OS) 
high-PT leptons with P| > 20 GeV/c (Iry^l < 1.0), at 
least two jets with measured Ei^* > 10 GeV (|?/""^*| < 
2.0), and > 25 GeV. To ensure that the is not 
due to mismeasurements, events were also required to 
have > 50 GeV, if the azimuthal angle between the 
direction of and the nearest lepton or jet was less 
than 20°. In order to enhance the purity of the sample a 
further cut was applied; Ht, the scalar sum of P|, £:f * 
and was required to exceed 170 (200) GeV in Run 
1 (Run 2) analysis d 0. Using 109 ±7 pb-^ of Run 1 
data, CDF found, in total, 8 events: ee, 1 /Lt/x and 7 e/i 
when, for example, 3.2 events was expected (top plus SM 
background [2^) in the e/i channel j^l- In Run 2, using 
193 pb'^ of data, CDF-II observed 13 events: 1 ee, 3 fifi 
and 9 e/i when the expected numbers, scaled to 13, are 
3.3 ± 0.5, 2.8 ± 0.5 and 6.8 ± 0.8 0. An excess in the e/i 
channel and a deficit in the ee channel have been found 
in both Run 1 and Run 2 counting experiments. We 
believe that this asymmetry does not look easy to explain 
simply in terms of different efficiencies, acceptances and 
backgrounds for the three classes of events. 
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TABLE II: Branching ratios for t ^ cS'^, S° ^ rfj, and 
for t — > cur. In models with strong dynamics, DFV decays 
proceed through a direct contact interaction. 

B. Model Independent Parametrization 

In the context of the previously discussed theoretical 
scenarios, MTV top decay t cfir are available with 
branching ratios up to 0.2. Br{t uTfi) is generally sup- 
pressed, except for the case of strongly interacting model, 
where both the branching ratios may reach similar val- 
ues. If B is the branching ratio for MTV top decays: 
B = Br{t — > c{u)^t) and we assume no other significa- 
tively accessible decay channels, either than the SM and 
the MTV itself: Br{t Wb) = 1-B. The possible decay 
modes for a tt pair are then: i) purely SM decays occur- 
ing with a branching ratio of (1 — S)^; ii) mixed SM-MTV 
decays accessible with a branching ratio of 2B{1 — B); Hi 
purely MTV top decays having a branching ratio of: B^. 
In total (1 - + 26(1 -B) + B^ = 1. Then SM-MTV 
tt decays, with one leg decaying into bWe and the other 
one into c/ir/i, may fake the SM dilepton signature. 
We found that a, B ~ 0.10 -f- 0.25 is able to provide an 
excess in the e/i channel and a deficit in the number of ee 
events that is consistent with CDF Run 1 and Run 2 data. 
Anyhow, we believe that, at the present, dilepton analysis 
cannot rule out any of the model studied in present paper. 

V. CONCLUSIONS AND PERSPECTIVES 

Triggered by CDF top dilepton analysis, that observed 
an excess in the number of e/z and a deficit in the number 
of ee top dilepton candidates, we studied the possibility 
that double flavor-violating top decays: t c{u)Tfi may 
contaminate the top sample. If MSSM predicts values 
for Br(t cTn) up to about 10~^ and Flavor Models 
gives ~ 4 X 10~2, Strongly Interacting Models offer the 
possibility to reach ~ 0.10 0.20. We found that such a 
value can explain the pattern observed in the data. As 
at present, top dilepton analysis cannot rule out any of 
the model discussed above, further studies will be needed 
in order to determine whether such new physics effects 
will be confirmed at Tevatron or may show up at lower 
branching ratios in next coming LHC experiments. 
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Typically, a collider experiments use cylindrical coordi- 
nate system about the beam axis Et = EsinO, Pt = 



PsinO, 



V ■■ 



-lntan(6l/2) and^T 



is the unit vector, in the azimuthal plane, that points to 
the j-th calorimeter tower. 

Main backgrounds are Drell-Yan {Z* /'y ee, fifi), 
Z ^ rr and WW production. Other sources are pro- 
cesses with a real lepton and a jet or a track faking a 
second lepton and processes in which mismeasured muon 
tracks can result in an overstimate of the_^T. 



